Jenkins Workshop

Pipelines & Blue Ocean 101

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License .

Have you used this?

Nostalgie 1

And this?

Nostalgie 2

And…​. this?

Nostalgie 3

Hello!

Olivier VERNIN

overnin

Damien DUPORTAL

damien
Presentation CloudBees

And you?

commitstrip

  • Developers?

  • Who is doing Continuous Integration/Delivery/Deployment?

    • Which tools?

      • Jenkins "Old School" ? / "New Generation" ?

      • GitLab ? TeamCity ? TravisCI ? Bamboo ?

      • Something Else?

Workshop

Goals

  • It is a "101" (starter) about:

    • Jenkins (Declarative) Pipelines

    • Blue Ocean

  • It is a Workshop: learn by "doing"

    • We start from scratch

    • Come back home with a fully working example

At Home: Getting the Workshop Lab as a local VM

  • You need Vagrant (2.1.0+) and VirtualBox (5.2.12+)

  • Run the following commands to start the local VM:

# Getting the VM template (1 time only)
vagrant box add jenkins-lab-workshop https://github.com/oufti-playground/lab-vm<...>
# Initialize the environment (1 time only)
mkdir ~/jenkins-lab-workshop && cd ~/jenkins-lab-workshop
vagrant init -m -f jenkins-lab-workshop
# Start the lab
vagrant up

On-Site: Getting the Workshop Lab as a Cloud VM

  • We provide you a set of Cloud VMs: only web browser is required

  • Access the "Instance Sheet Page", put your name on the "Name" column to allocate the instance for yourself, and click on the "Instance URL" link on the same line.

  • Welcome to the Lab Home Page!

Accessing Customized Slides

  • Each instance hosts a copy of this slide-deck, with customized links

  • If you click the link "Workshop Slides", a new tab will open, to this slide-deck

Accessing Source Code

Accessing Jenkins

Jenkins and Blue Ocean

Jenkins

jenkins logo

  • Task orchestrator

  • Open-Source software written in Java

  • One of the first and most popular Continous Integration Engine

  • Plugin based

Jenkins Pipelines

pipeline

  • A tool to express your software pipeline with a DSL

  • Defined into a text file: Jenkinsfile

  • "Pipeline-as-code": stored into your SCM

  • != "Job DSL" (different areas)

Benefits of Jenkins Pipelines

  • Follows from your code workflows: branches, pull request, code review, even trunk-based

  • Reduces the number of jobs

  • A single execution ("build") can run on multiple machines

  • Durable: survives a Jenkins restart

Getting started with Pipeline

i have no idea what i am doing
BlueOcean

Blue Ocean

  • A new developer experience for using Jenkins

  • A modern GUI written in React, in parallel of the "Classic UI"

  • Pipeline-centric

  • Install it with a plugin

Your First Pipeline

Blue Ocean

  • We want to create a Pipeline from scratch:

    • Documentation is hard. Let’s try Blue Ocean.

  • Browse to your Jenkins instance and switch to Blue Ocean:

Follow along: Initialize Your First Pipeline 1/4

  • Follow the presenter instructions to create a new Pipeline:

    • Stored in Git

    • Get the SSH URL from the git server

    • Configure Jenkins access to the source code:

      • In Gitea, click the top-right user drop down

      • Browse to SSH/GPG Keys

      • Add a new key, by copy and pasting from Jenkins

Blue Ocean Pipeline Editor

  • Provides a Pipeline’s graphical editor in Blue Ocean, with full round trip with SCM:

    • Scaffold your Pipeline, Save it, Execute it.

  • Supports existing Declarative Pipeline, as starting from scratch

  • Requires a compatible SCM (Git over SSH, Github, BitBucket, etc.)

Follow along: Initialize Your First Pipeline 2/4

  • Blue Ocean cannot find any Jenkinsfile

    • It opens automatically the Pipeline Editor for you

  • A red message appears saying: "a stage is needed":

    • Create a stage named Build by clicking the big "+"

  • Each stage (virtual) is composed of "steps" (implementation)

    • "At least one step is required"

Follow along: Initialize Your First Pipeline 3/4

  • We want the stage Build to have 2 steps:

    • Print the message Building…​

    • Execute the shell script ./scripts/build.sh

  • Time to run this Pipeline: click the "Save" button

    • Add a meaningful Description (commit message)

    • Commit to 'master'

    • Clic Save & Run, and see the build kicking off

Follow-along: Initialize Your First Pipeline 4/4

  • Take time to browse the build page in Blue Ocean

    • Click on the stage, unfold the logs…​

  • Let’s have a look to the generated Jenkinsfile in the SCM:

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        echo 'Building...'
        sh './scripts/build.sh'
      }
    }
  }
}

Exercise: Add 2 New Stages to your First Pipeline

  • Goal: add 2 new stages to your Pipeline:

    Integration Tests

    runs the script ./scripts/integration-tests.sh

    Deploy

    runs the script ./scripts/deploy.sh

  • Use the Blue Ocean Pipeline Editor:

    • Click on the "Edit" button (pen’s icon)

    • Use the Build stage as example

Solution: Add 2 New Stages to your First Pipeline

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        echo 'Building...'
        sh './scripts/build.sh'
      }
    }
    stage('Test') {
      steps {
        sh './scripts/integration-tests.sh'
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Pipeline: Tests and Webhooks

Follow-along: Test Reports with JUnit 1/3

  • Junit is …​.

  • We want to report test results to Jenkins

  • Let’s start with Unit Test’s results

    • Generated during the Build stage

    • XML files located in target/surefire-reports/

Follow-along: Test Reports with JUnit 2/3

  • Open the Pipeline in Blue Ocean Editor

  • Edit the stage Build

  • Remove the Print Message step

  • Add a new Archive JUnit-formatted test results step

    • Set the field TestResults to target/surefire-reports/*.xml

  • Save the pipeline and check the build

Follow-along: Test Reports with JUnit 3/3

  • When the build finished, check the "Tests" section

  • Let’s have a look to the generated Pipeline:

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        sh './scripts/build.sh'
        junit 'target/surefire-reports/*.xml'
      }
    }
    stage('Test') {
      steps {
        sh './scripts/integration-tests.sh'
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Exercise: Integration Tests Report

  • We also want to publish integration tests reports

    • Generated during the Integration Tests stage

    • XML files located in target/failsafe-reports/

  • Expected status of the build: UNSTABLE

Solution: Integration Tests Report

  • What are the changes betwee UNSTABLE and SUCCESS?

    • Color, Test failure notification badge, etc.

Continuous Integration (CI)

fail fast continuous integration

  • Each code integration is validated by an automated build

  • Code is integrated often, at least daily, to be a non-event

  • Feedback loop to close the continous cycle

Exercise: Réparer le build

  • Le build est en état UNSTABLE (jaune)

  • Priorité: réparer le build

  • Utiliser Gitea git server

  • Les tests d’intégration sont dans src/test/java/hello

  • Indices:

    • Integration Tests: == IT

    • Il suffit de savoir commenter/dé-commenter

Solution: Réparer le build

  • Depuis src/test/java/hello/HelloControllerIT.java

    • Cliquer sur "Edit"

  • Commenter la ligne 39

  • Dé-commenter la ligne 40

  • Commit avec un message (push automatique)

  • Lancer le build manuellement dans Blue Ocean

  • Le build doit être vert (Stable)

Webhooks pour un retour plus rapide

  • Nous avons dû lancer le build manuellement

  • IC: Retours rapides !

    • Lancer le build dès que le code est poussé

Exercise: Webhooks

Solution: Webhooks

  • Ajoutez un commentaire dans le Jenkinsfile depuis Gitea git server

    • Un build va démarrer

    • Validez dans l’onglet "Changes"

Aller plus loin…​

  • Dans l’éditeur Blue Ocean, voir la version textuelle:

    • Combinaison CTRL + S (On Mac: CMD +S)

    • Bi-directionnel: essayez de charger une solution de pipeline

  • Le Pipeline Syntax Snippet Generator comme acolyte:

Agents

Qu’est ce qu’un Agent?

  • Un noeud (ou node) est une machine prête à recevoir des builds

  • Step agent spécifie sur quel "noeud" exécuter des "stages".

  • Une section agent globale doit être définie (au niveau du block pipeline)

  • On peut aussi définire des sections agent par "stage"

Exercice - Agents

  • Exécuter l’étape Build sur un agent configuré avec le label maven-jdk8

  • Exécuter l’étape Test sur un agent configuré avec le label java8

  • L’éditeur Blue Ocean est utilisable

Solution - Agents

pipeline {
  agent any
  stages {
    stage('Build') {
      agent {
        node {
          label 'maven-java8'
        }
      }
      steps {
        sh './scripts/build.sh'
      }
      post {
        always {
          junit 'target/**/*.xml'
        }
        success {
          archiveArtifacts 'target/*.jar'
          stash(name: 'build-result', includes: 'target/**/*')
        }
      }
    }
    stage('Test') {
      agent {
        node {
          label 'java8'
        }
      }
      steps {
        unstash 'build-result'
        sh './scripts/test.sh'
        junit 'target/**/*.xml'
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Exercice - Tests Parallèles

  • But: Tester en parallèle l’application sur java 7 et 8

  • Mot clef parallel définissant un block contenant des "stages"

  • Agent java7 pour le Test Java 7

  • L’éditeur Blue Ocean est utilisable (et recommandé)

Solution - Tests Parallèles

pipeline {
  agent any
  stages {
    stage('Build') {
      agent {
        node {
          label 'maven-java8'
        }
      }
      steps {
        sh './scripts/build.sh'
      }
      post {
        always {
          junit 'target/**/*.xml'
        }
        success {
          archiveArtifacts 'target/*.jar'
          stash(name: 'build-result', includes: 'target/**/*')
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Test Java 8') {
          agent {
            node {
              label 'java8'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
        stage('Test Java 7') {
          agent {
            node {
              label 'java7'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Agents avec Docker

  • But: Usage de Docker pour faciliter la définition des environements de build

  • Le mot clef agent permet d’exécuter les "stages" dans un container Docker, depuis une "image Docker", ou depuis un Dockerfile (recette maison d’image Docker)

Exercice - Agent Docker

  • Exécuter le Build dans un containeur basé sur le fichier Dockerfile.build

  • Exécuter le Test Java 8 dans un containeur basé sur les images maven:3-jdk-8-alpine

  • Trick: documentation manquante sur filename, dans un block dockerfile

Solution - Agent Docker

pipeline {
  agent any
  stages {
    stage('Build') {
      agent {
        dockerfile {
          filename 'Dockerfile.build'
          label 'docker'
        }
      }
      steps {
        sh './scripts/build.sh'
      }
      post {
        always {
          junit 'target/**/*.xml'
        }
        success {
          archiveArtifacts 'target/*.jar'
          stash(name: 'build-result', includes: 'target/**/*')
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Test Java 8') {
          agent {
            docker {
              image 'maven:3-jdk-8-alpine'
              label 'docker'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
        stage('Test Java 7') {
          agent {
            node {
              label 'java7'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Pipeline Avancés

Input

  • Jenkins permet de "mettre en pause" l’exécution d’un pipeline, en attendant une validation humaine

  • Continuous Delivery / Deployment

  • Mot-clef input, c’est une "step" Pipeline

  • Généralement utilisé dans un "stage" dédié.

    • Obligatoirement avec agent none (pas d’exécuteur, pas de workspace)

Exercice - Validation Humaine

  • Ajouter une stage "approval" avant le déploiement

Solution - Validation Humaine

pipeline {
  agent any
  stages {
    stage('Build') {
      agent {
        dockerfile {
          filename 'Dockerfile.build'
          label 'docker'
        }
      }
      steps {
        sh './scripts/build.sh'
      }
      post {
        always {
          junit 'target/**/*.xml'
        }
        success {
          archiveArtifacts 'target/*.jar'
          stash(name: 'build-result', includes: 'target/**/*')
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Test Java 8') {
          agent {
            docker {
              image 'maven:3-jdk-8-alpine'
              label 'docker'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
        stage('Test Java 7') {
          agent {
            node {
              label 'java7'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
      }
    }
    stage('Approval') {
      agent none
      steps {
        input(message: 'Deploy Application?', ok: 'Yes Deploy!')
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
    }
  }
}

Conditionnal Stage

  • Définir l’exécution conditionnelle d’une "stage"

  • Mot clef when

  • Doit contenir au moins une condition parmi:

    • branch

    • environment

    • expression

    • Logique "built-in": allOf, anyOf, etc.

  • When Documentation

Variables d’environments

  • Définir des collections de clé-valeurs, en tant que variables d’environement:

    • Définition globale ou par "stage"

    • Ou en utilisant withEnv dans un block steps pour une instruction spécifique

Exercice - Déploiement Conditionnel

  • N’éxecuter les "stages" Approval et Deploy que:

    • Si on se trouve sur la branche master

    • Ou si la variable FORCE_DEPLOY est à true

Solution - Déploiement Conditionnel

pipeline {
  agent any
  environment {
    FORCE_DEPLOY = 'false'
  }
  stages {
    stage('Build') {
      agent {
        dockerfile {
          filename 'Dockerfile.build'
          label 'docker'
        }
      }
      steps {
        sh './scripts/build.sh'
      }
      post {
        always {
          junit 'target/**/*.xml'
        }
        success {
          archiveArtifacts 'target/*.jar'
          stash(name: 'build-result', includes: 'target/**/*')
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Test Java 8') {
          agent {
            docker {
              image 'maven:3-jdk-8-alpine'
              label 'docker'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
        stage('Test Java 7') {
          agent {
            node {
              label 'java7'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
      }
    }
    stage('Approval') {
      agent none
      steps {
        input(message: 'Deploy Application?', ok: 'Yes Deploy!')
      }
      when {
        anyOf {
          branch 'master'
          environment name: 'FORCE_DEPLOY', value: 'true'
        }
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
      when {
        anyOf {
          branch 'master'
          environment name: 'FORCE_DEPLOY', value: 'true'
        }
      }
    }
  }
}

Paramètres

  • Directive parameters : l’utilisateur fournit des argument au pipeline

  • Disponible en tant que variables accessibles dans l’objet params

  • Oeuf & Poule : Jenkins ne peut accéder au paramètre lors dur 1er build

Exercice - Paramètres

  • Ajouter un paramètre DEPLOY_MESSAGE dont la valeur par défaut est Deploy ?

  • Ce paramètre est utilisé dans le input

Solution - Paramètres

pipeline {
  agent any
  parameters {
    string(name: 'DEPLOY_MESSAGE', defaultValue: 'Deploy ?', description: 'Message de déploiement')
  }
  environment {
    FORCE_DEPLOY = 'false'
  }
  stages {
    stage('Build') {
      agent {
        dockerfile {
          filename 'Dockerfile.build'
          label 'docker'
        }
      }
      steps {
        sh './scripts/build.sh'
      }
      post {
        always {
          junit 'target/**/*.xml'
        }
        success {
          archiveArtifacts 'target/*.jar'
          stash(name: 'build-result', includes: 'target/**/*')
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Test Java 8') {
          agent {
            docker {
              image 'maven:3-jdk-8-alpine'
              label 'docker'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
        stage('Test Java 7') {
          agent {
            node {
              label 'java7'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
      }
    }
    stage('Approval') {
      agent none
      steps {
        input(message: params.DEPLOY_MESSAGE, ok: 'Yes Deploy!')
      }
      when {
        anyOf {
          branch 'master'
          environment name: 'FORCE_DEPLOY', value: 'true'
        }
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
      when {
        anyOf {
          branch 'master'
          environment name: 'FORCE_DEPLOY', value: 'true'
        }
      }
    }
  }
}

Options & Configurations

  • Options: Configurer un "job" depuis le Pipeline

  • Configuration: Même chose dans la GUI "Legacy"

  • Mot clef options

  • Documentation des options: https://jenkins.io/doc/book/pipeline/syntax/#options

  • Certaines options comme timeout peuvent être appliquée dans un bloc step

Exercice - Options & Timeout Approval

  • On souhaite que la "stage" Approval attende 3 minute avant d’arrêter le pipeline

  • On ne veut conserver que les 5 derniers builds d’un pipeline: limiter l’usage disque

Solution - Options & Timeout Approval

pipeline {
  agent any
  options {
    buildDiscarder(logRotator(numToKeepStr: '5'))
  }
  parameters {
    string(name: 'DEPLOY_MESSAGE', defaultValue: 'Deploy ?', description: 'Message de déploiement')
  }
  environment {
    FORCE_DEPLOY = 'false'
  }
  stages {
    stage('Build') {
      agent {
        dockerfile {
          filename 'Dockerfile.build'
          label 'docker'
        }
      }
      steps {
        sh './scripts/build.sh'
      }
      post {
        always {
          junit 'target/**/*.xml'
        }
        success {
          archiveArtifacts 'target/*.jar'
          stash(name: 'build-result', includes: 'target/**/*')
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Test Java 8') {
          agent {
            docker {
              image 'maven:3-jdk-8-alpine'
              label 'docker'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
        stage('Test Java 7') {
          agent {
            node {
              label 'java7'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
      }
    }
    stage('Approval') {
      agent none
      steps {
        timeout(time: 3, unit: 'MINUTES') {
          input(message: params.DEPLOY_MESSAGE, ok: 'Yes Deploy!')
        }
      }
      when {
        anyOf {
          branch 'master'
          environment name: 'FORCE_DEPLOY', value: 'true'
        }
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
      when {
        anyOf {
          branch 'master'
          environment name: 'FORCE_DEPLOY', value: 'true'
        }
      }
    }
  }
}

Shared Libraries

Exercice - Shared Libraries

  • Create a repository jenkins-libs

    • Init with a file vars/runPipeline.groovy:

      def call(String MESSAGE) {
  echo "Lib: ${MESSAGE}"
}

  • Switch to legacy UI, configure the Multibranch job

    • Add the shared library

  • Adapt the pipeline to run the function in Build stage

Solution - Shared Libraries

pipeline {
  agent any
  libraries {
    lib('jenkins-libs@master')
  }
  options {
    buildDiscarder(logRotator(numToKeepStr: '5'))
  }
  parameters {
    string(name: 'DEPLOY_MESSAGE', defaultValue: 'Deploy ?', description: 'Message de déploiement')
  }
  environment {
    FORCE_DEPLOY = 'false'
  }
  stages {
    stage('Build') {
      agent {
        dockerfile {
          filename 'Dockerfile.build'
          label 'docker'
        }
      }
      steps {
        runPipeline('Salut')
        sh './scripts/build.sh'
      }
      post {
        always {
          junit 'target/**/*.xml'
        }
        success {
          archiveArtifacts 'target/*.jar'
          stash(name: 'build-result', includes: 'target/**/*')
        }
      }
    }
    stage('Test') {
      parallel {
        stage('Test Java 8') {
          agent {
            docker {
              image 'maven:3-jdk-8-alpine'
              label 'docker'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
        stage('Test Java 7') {
          agent {
            node {
              label 'java7'
            }
          }
          steps {
            unstash 'build-result'
            sh './scripts/test.sh'
            junit 'target/**/*.xml'
          }
        }
      }
    }
    stage('Approval') {
      agent none
      steps {
        timeout(time: 3, unit: 'MINUTES') {
          input(message: params.DEPLOY_MESSAGE, ok: 'Yes Deploy!')
        }
      }
      when {
        anyOf {
          branch 'master'
          environment name: 'FORCE_DEPLOY', value: 'true'
        }
      }
    }
    stage('Deploy') {
      steps {
        sh './scripts/deploy.sh'
      }
      when {
        anyOf {
          branch 'master'
          environment name: 'FORCE_DEPLOY', value: 'true'
        }
      }
    }
  }
}